Repository navigation
Conversation
ShimmerBluetooth.IOThread and LiteProtocol.IOThread ran tight while(!stop) loops with no sleep, calling processNextInstruction()/bytesAvailableToBeRead() as fast as possible, pegging a CPU core per connected device even when idle. Track whether an iteration did useful work (instruction processed, actively streaming, or bytes read) and only sleep briefly (2ms) when it didn't - latency of the active/streaming paths is unchanged since sleep is skipped whenever there's real work to do. ShimmerBluetooth.ProcessingThread spun on mABQPacketByeArray.isEmpty(); since the queue is an ArrayBlockingQueue, swap to a blocking poll(100ms) instead so the thread parks while idle rather than busy-checking emptiness. Also replace two teardown busy-waits with bounded Thread.join() calls so closing a connection doesn't spin a core while waiting for the reader threads to exit: ShimmerPC.closeConnection() (mIOThread + mPThread) and LiteProtocol.stopIoThread() (mIOThread). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JjgNkS6wtPkQUdK3KdHVdN
…un(), interrupt-exit loops Review findings on the idle-sleep changeset: - LiteProtocol.IOThread.run() was declared synchronized; Thread.join() locks the same Thread instance, so external stopIoThread() callers blocked on monitor entry with no timeout until the loop exited, defeating the bounded join. run() is no longer synchronized. - stopIoThread()/closeConnection() can be reached synchronously from the reader thread itself (killConnection from IOThread's catch; connectionLost on serial errors), where join() can never succeed and just burns the full 2s timeout. Skip the join when running on that thread. - Reader loops now also exit on thread interrupt: previously an interrupt without stop=true would make poll()/sleep() throw immediately every iteration - a latent busy-spin. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JjgNkS6wtPkQUdK3KdHVdN
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
processNextInstruction()'s no-instruction branch already waits 50ms (threadSleep(50)) when idle (no instruction pending, not streaming, no bytes available). The new outer idle-sleep in IOThread.run() added a further Thread.sleep(2) whenever didWork stayed false, so the fully idle case waited 52ms instead of 50ms. Fix: have that branch report true (i.e. "waited") instead of false, so run()'s outer idle-sleep is skipped for the iteration that already paced itself. This is the minimal change - it keeps the pre-existing 50ms idle cadence exactly as before this PR, rather than replacing it with the new 2ms cadence (a bigger behavioural change) or leaving the two sleeps stacked. The 50ms sleep only fires when there is no instruction on the stack at all, so it has no bearing on pacing retries for an in-flight instruction - removing it in favour of the 2ms sleep was rejected because that would make idle polling 25x more frequent than before. Applied identically to ShimmerBluetooth.java and LiteProtocol.java, with matching javadoc updates on processNextInstruction(). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
…cessingThread self-join, volatile stop Round-2 review findings on the bounded-join teardown from the previous commit: - ShimmerPC.closeConnection() and ShimmerGRPC.closeConnection() are unsynchronized and can run concurrently (connectionLost() on the IOThread itself vs. a user-triggered disconnect() on another thread). Both dereferenced mIOThread/mPThread after only a non-atomic null check, so a concurrent caller nulling the field out mid-method could turn the passed check into an NPE - swallowed by the outer catch(Exception), skipping purgePort/closePort and leaking the COM port. Fixed by snapshotting mIOThread/mPThread into locals once and operating on the snapshot throughout. - Added the same self-join guard used for mIOThread to mPThread in both classes: buildAndSendMsg() runs on the ProcessingThread and dispatches to app listeners, so a listener calling disconnect() would otherwise self-join and burn the full 2s timeout. - After each bounded join(2000), if the thread is still alive, log a warning and call interrupt() so the thread's own interrupt checks can finish tearing it down, instead of silently closing the port under a still-live reader. Applied to mIOThread/mPThread in ShimmerPC.closeConnection() and ShimmerGRPC.closeConnection(), and to mIOThread in LiteProtocol.stopIoThread(). - ShimmerGRPC.closeConnection() previously did an unbounded join() with no self-join guard; converted to the same bounded snapshot+guard+alive-check pattern, keeping its existing interrupt() calls. - Made the `stop` flags volatile (ShimmerBluetooth.IOThread, ShimmerBluetooth.ProcessingThread, LiteProtocol.IOThread) so a writer on one thread is visibly seen by the reader thread without relying on incidental synchronization elsewhere. - Softened the processNextInstruction() idle-branch comment in ShimmerBluetooth.java: getInstruction() returning null doesn't only mean "no instruction pending" - it also happens right after removing a leading null placeholder, with a real instruction possibly next in the list. No behavioural change, comment only. Re-verified: streaming path in both IOThread.run() loops still never sleeps, every idle path still sleeps (no spin), and closeConnection() still reaches purgePort/closePort on every path. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
…errupt before join, tidy-ups Copilot review findings on PR #279: - ShimmerBluetooth.java and LiteProtocol.java: the idle-branch sleep in processNextInstruction() used threadSleep(50), whose shared implementation swallows InterruptedException without restoring the interrupt flag - defeating run()'s !Thread.currentThread().isInterrupted() exit check. Replaced with a local try/Thread.sleep(50)/catch that re-sets the flag via Thread.currentThread().interrupt(). Left every other threadSleep() call in both files untouched. - ShimmerBluetooth.java IOThread.run(): fixed inconsistent indentation in the InShimmerTest branch (didWork assignment and the readBytes block were not nested under the bytesAvailableToBeRead() guard). Indentation only - control flow verified unchanged. - ShimmerPC.java: fixed a duplicated "before before" in a comment. - ShimmerPC.closeConnection(), ShimmerGRPC.closeConnection(), and LiteProtocol.stopIoThread(): interrupt() is now called before the bounded join(2000) (still only in the non-self-join branch) instead of only after a failed join. Since run() now exits on Thread.currentThread().isInterrupted() and any idle Thread.sleep()/ queue poll() wakes immediately on interrupt, this makes shutdown faster and more deterministic. The post-join "still alive" branch now just logs a warning instead of interrupting a second time. - ShimmerGRPC.closeConnection(): moved the pre-existing interrupt() calls (for both mIOThread and mPThread) inside the Thread.currentThread() != t guard - previously they ran unconditionally and would mark the *current* thread interrupted when closeConnection() runs on the IOThread/ProcessingThread itself (the self-close case), which is neither needed nor correct. Safety check for interrupting a thread blocked in a serial read (requested before making the interrupt-before-join change): ShimmerPC reads via ByteCommunicationJSSC, which calls JSSC's SerialPort.readBytes(count, timeout) - a native, timeout-bounded JNI call that is not interruptible and never throws InterruptedException; Thread.interrupt() on a thread blocked there only sets the interrupt flag, observed once the call returns (by timeout or data) at the top of run() or in a subsequent Thread.sleep()/queue.poll(). bytesAvailableToBeRead()/readBytes()/ writeBytes() only catch SerialPortException/SerialPortTimeoutException/ NullPointerException, none of which interrupt() can trigger, so no new connectionLost() -> closeConnection() recursion is introduced. ShimmerGRPC reads via a ThreadSafeByteFifoBuffer whose read() does declare InterruptedException and is already caught locally (printed and treated as "no data"), so it is safe by construction. No path was found where the interrupt flag causes a new failure mode; the change was made in all three teardown sites as requested. Re-verified: both IOThread.run() loops still never sleep on the streaming path, no path can spin without sleeping, and closeConnection() still reaches purgePort/closePort on every path. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
- LiteProtocol.stopIoThread(): snapshot mIOThread into a local and use it for stop/interrupt/join/isAlive, so a concurrent teardown that nulls the field after the null check can no longer cause an NPE. Only clear the field if it still points at the thread that was stopped, so a newly started IOThread is not clobbered. - ShimmerPC/ShimmerGRPC.closeConnection(): move the ProcessingThread (mPThread) teardown out of the `if (ioThread != null)` block into its own sibling block, so it still runs when another caller has already nulled mIOThread while the ProcessingThread is alive. The logic of each block is otherwise unchanged. The merge of origin/master is in the preceding merge commit. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
F1: closeConnection() closes the serial port after a bounded 2 s join even if the IOThread is still blocked in a non-interruptible jssc call. When that call then failed on the closed port (or NPE'd on the nulled mByteCommunication), the ShimmerPC I/O helpers called connectionLost(), setting CONNECTION_LOST after a deliberate disconnect, which AbstractConsensysApi answers with an auto-reconnect in NeuroLynQ. The catch sites in bytesAvailableToBeRead/availableBytes/writeBytes/readBytes, and ShimmerGRPC.writeBytes, now skip connectionLost() when running on an IOThread whose stop flag is set. Both closeConnection()s also clear mIOThread/mPThread only if they still hold the thread that was stopped, as LiteProtocol.stopIoThread() already does, so a late closeConnection() cannot clobber a reconnect's new threads. F2: LiteProtocol.IOThread.run() no longer calls killConnection() when stop is already set: the teardown interrupt surfacing from txBytes as ERR_WRITING_DATA caused a spurious error event and a re-entrant disconnect. Test: API_00028_IoThreadTeardownTest (ShimmerDriverPC) pins that idle IOThread/ProcessingThread use <300 ms CPU per second, that disconnect() stops both within 1500 ms, and that a write blocked past the join and then failing on the closed port leaves the device DISCONNECTED (fails without F1). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
Live time-axis plots moved their X window from the data thread after every sample (a PlotCustomFeature calling setXAxisRangeBasedOnXDuration(), which also allocated a new RangePolicyFixedViewport 256 times a second). The window's right edge was the newest sample's X, so it advanced in Bluetooth bursts of 10-30 samples, and Chart2D's 50 ms repaint timer showed each burst as a jump - worst at UHD, where a jump covers twice the pixels. BasicPlotManagerPC now has an opt-in frame clock (setLiveViewportFrameClockEnabled): one javax.swing.Timer per plot manager, FRAME_INTERVAL_MS = 16 (33 is the documented fallback if 60 fps is too heavy on the EDT). Each tick, on the EDT, advances the window's right edge by the real time elapsed since the previous tick and repaints the chart directly, so each frame shows exactly one even step. While it runs, Chart2D's own repaint timer is backed off to 250 ms (restored on stop) so it does not add a second, unsynchronised stream of paints. Latency design: the right edge is a playout clock that trails "newest X + time since that sample arrived" by VIEWPORT_LATENCY_MS = 200, which absorbs the bursts and the IOThread -> callback -> plot-thread hops. It is steered toward that target by 5% of the error per tick, each correction capped at 10% of the tick's elapsed time, snaps when more than 1 s off, never runs past the newest sample (a stall freezes the window, as before) and never scrolls backwards. It is deliberately not anchored to System.currentTimeMillis(): System_Timestamp_Plot is the device clock offset once onto the PC epoch at the first packet, so it carries that packet's latency and drifts with the device crystal, and the wall clock can be stepped by NTP; System.nanoTime() plus the newest sample needs no common epoch. Points newer than the right edge are clipped by jchart2d, not drawn into the margin. Lifecycle (DEV-895/DEV-717): the clock starts when live time-axis samples arrive into a showing chart, and stops itself when none has arrived for 2 s, the plot is paused, the chart is not showing, the X axis is not the time axis, or the data was cleared; removeAllSignals() (the frame-close path) stops it immediately. On stop the window is left ending at the newest sample. A stopped Swing Timer is not referenced by the TimerQueue, so it does not pin the frame. The tick takes only the chart monitor, never the list monitors, so the list -> chart lock order is kept. mCurrentXValue is now volatile; the data thread's per-sample cost is a nanoTime() read and a volatile write. setXAxisRangeBasedOnXDuration() and its sampling-rate variant are unchanged for explicit callers (ConsensysGQ's right-aligned plots, PlotEventMarkers); frequency/FFT and value X axes never run the clock. VISIBLE CHANGE: DEFAULT_LINE_THICKNESS is now 1 (was 2), so live traces are thinner. A full paint of 2560 points into a 3600x1800 chart measured 9.2 ms at 2 px against 2.3 ms at 1 px. UtilChart2D's separate constant is left at 2: only QuickPlot uses it, not live traces. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
…device X Follow-up to 2e22633 after review. - Catch-up: below the 1 s snap threshold the playout clock could only speed up by 10% of real time, so a 600-1100 ms stall or a 500 ms forward X jump left the newest data hidden for 3-8 s, and the lag could outgrow the 0.4 s trace margin. The correction is now asymmetric: slowing down and errors under 50 ms keep the 10% cap and 5% gain; from 50 ms to 200 ms behind, the cap ramps linearly to 100% (2x real-time scrolling) and the gain to 15%. Simulated recovery to <260 ms hidden: 300/600/900/1100 ms stalls 68/376/ 668/836 ms, +500/+900 ms X jumps 576/976 ms; no backward steps, no overshoot, max step 32 ms per 16 ms tick. The 1 s snap is unchanged. - Stroke: revert DEFAULT_LINE_THICKNESS to 2, so reports, GQ/NeuroLynQ, PlotEventMarkers, PanelCalibrateShimmer and every other plot keep 2 px. New per-manager setTraceLineThickness(float), defaulting to DEFAULT_LINE_THICKNESS, is used by createNormalTrace() and by setTraceLineStyle() for CONTINUOUS/INDIVIDUAL_POINTS (which previously re-applied the static). The comment now states what jchart2d does (TracePainterPolyline, i.e. Graphics.drawPolyline) and the measured cost of a 2560-point UHD paint: ~0.4 vs ~3.7 ms for 1 vs 2 px at identity transform, ~3.5 vs ~4.0 ms at uiScale 2. - First frame: startFrameClockOnEdt() now runs one tick synchronously, so a chart shown again does not paint its stale window for the first frame. - Multi-device plots: mCurrentXValue alternates between the devices' timestamps, so with a >=1 s offset the snap branch made the window jump back and forth, and with smaller offsets it followed the slower device. The clock now uses mFrameClockNewestX (and the arrival time of that X), kept by the data thread as max(X); it restarts from a lower X only on a genuine backward jump: more than 1 s behind with no sample near the newest for 500 ms (without that confirmation a 1300 ms device offset still flipped it on every sample). Reset to 0 with mCurrentXValue in removeAllSignals() and clearAllDataBuffer(). Idle detection still uses any device's last sample. mCurrentXValue itself is unchanged. Simulated 0-1300 ms offsets: zero backward steps, hidden lag vs the faster device mean 172 ms (144-201); a -5 s jump re-syncs with one snap. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
Round-2 review follow-up to 5aeba04. The playout clock assumes X advances ~1 ms per wall-clock ms and that new X arrives well within VIEWPORT_LATENCY_MS. Two real cases break that: - Consensys DB playback runs recorded data through the same live plots; PlaybackSession's timer period is truncated, so X runs ~1.3x at "x1" and 2-4x+ at x2/x4. The 2x catch-up cap could not keep up, the 1 s resync snapped forward 17-82 times per 25 s and a blank strip opened at the left. - Low-rate series (the 2 Hz fusion-response plot, any device < 5 Hz) froze at the newest-sample clamp, then caught up at 2x (40% frozen ticks at 2 Hz). computeLiveViewportEnd() now estimates, on the EDT, the X rate (dNewestX/dArrival) and the spacing between new X values over windows of >= 500 ms of arrivals (so a stall followed by buffered data still reads as ~1x), EWMA-smoothed (alpha 0.5). If the rate is > 1.5 or < 0.67, or the spacing > 1.2 x VIEWPORT_LATENCY_MS, in both the window and the EWMA for two windows in a row, it switches to anchored mode: the right edge is the newest X, as before DEV-896 but still one update and one paint per tick. When anchored for fast X, each tick may advance at most 1.5x the estimated rate, so two playback deliveries landing in one 16 ms frame are spread over the following frames instead of drawing as one double step. It switches back when the rate is within 0.83-1.2 and the spacing < 0.75 x latency for two windows, then holds the window still until the playout target catches up (~110-130 ms) instead of stepping back. The estimate resets when the clock stops. The spacing threshold is 1.2x the latency, not 0.75x: a 5 Hz series still scrolls evenly in playout mode (0% frozen) and would only get jumpier anchored. Sims (25 s each): playback 2.6/3.9/5.2x go from 17/50/82 forward snaps to 0, with lag behind the newest X <= 37 ms (trace margin 400). At 1/2/3/4 Hz the longest frozen run is one sample interval, and the only fast ticks are the anchored jumps themselves. The stall, X-jump and two-device tables from the last round are unchanged. Also: - onFrameClockTick() stops without snapping when the X axis is no longer the time axis, so a value/frequency axis is not given a time window. - The clock also stops when the chart's top-level Frame is iconified (an iconified JFrame's components still report isShowing()), and is not started while it is. - New API_00029_LiveViewportFrameClockTest (Swing-free, synthetic nanoTime): steady stream, stall catch-up, long EDT tick, far-behind second device, genuine backward jump, X rate 4x (no step > 100 ms, lag within the trace margin), 2 Hz data (no frozen run longer than one sample interval). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
DEV-895 Stop per-device reader threads busy-spinning when idle
…n tests The simulator connect tests called connect() and then slept a fixed 3-5 s before asserting on the device. isConnected() turns true before the post-connect config read (infomem, calibration) runs, so on a loaded CI runner the read could miss the window and assertions such as getTrialDurationEstimatedInSecs() failed - seen in Shimmer-Advance-API run 37567210311, which passed on rerun. Add ByteCommunicationTestUtils.connectAndWaitUntilInitialised(), which waits until isInitialised() and the CONNECTED state (30 s ceiling, fails with a message on timeout), and use it for all 22 connect waits. The TaskCompletionSources it replaces were never completed, so they were only ever sleeps. Also disconnect the device in @after so earlier tests' IO threads and status timers no longer run on into later tests. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Lets the Shimmer3R SD log parser set the low-noise accel (LSM6DSV) range from the file header, as setLSM6DSVGyroRange does for the gyro. mSensorLSM6DSV is private, and ShimmerObject.setAccelRange sets the wide-range accel on Shimmer3R, so there was no way to reach it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…_accel_range DEV-1163 Add ShimmerObject.setLSM6DSVAccelRange for the Shimmer3R SD log parser
Wait for initialisation instead of a fixed sleep in byte-communication tests
Process.onExit() is Java 9+. ShimmerJavaClass.jar is built as Java 8 bytecode for MATLAB, and source/target 1.8 compiled this call without complaint, so starting the BLE gRPC server from MATLAB on Java 8 would throw NoSuchMethodError. The jar currently in Shimmer-MATLAB-ID/libs (built 2025-11-24) contains the call. Wait on the process in a daemon thread instead, which logs the same exit message. Found by building the jar with --release 8. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Actions -> "Release Java API" releases the version already in the build files (bumped by the usual uprev PR; master is protected): - tests and builds as gradle.yml does - publishes shimmerdriver, shimmerbluetoothmanager and shimmerdriverpc in one gradlew publish, so they cannot drift apart again - tags API_v<version> and creates the GitHub Release with the jars, not marked Latest - builds ShimmerJavaClass.jar as Java 8 bytecode on its own runner and opens a PR replacing Shimmer-MATLAB-ID/libs/ShimmerJavaClass.jar (needs the MATLAB_REPO_TOKEN secret) ShimmerDriverPC gains -PmatlabJava8, which compiles all three projects with --release 8 in place of hand-editing three build files to 1.8. Publishing URLs move from ShimmerEngineering, now a user account, to ShimmerResearch. AGENTS.md documents the release steps. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The version in the build files is normally one that has already been released until the next uprev, so stopping there made every PR dry run end before the build it is meant to test. Dry runs now warn; a real release still refuses. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The lns0_16_11_w_sd_bt_streaming connect hangs on Linux CI and not on Windows, so the CI log is the only evidence of where the IO thread is stuck. Print every thread's full stack, the monitors it holds, the lock it waits on and any JVM-detected deadlock before failing. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The connect sequence unlocks the instruction stack and then starts the read-status timer with no delay, so the timer's first readStatusTask (device lock, then instruction-list lock) runs just as the IOThread takes the first post-connect instruction in getInstruction() (list lock, then device lock). When they meet, both block for good and the connect never finishes. The thread dump from the previous commit shows that cycle on Linux CI in lns0_16_11_w_sd_bt_streaming, and locally in API_0000X_ByteCommunicationShimmer3. The driver is deliberately left unchanged. The simulator tests now build their ShimmerPC through ByteCommunicationTestUtils.newShimmerPCWithoutStatusTimer(), which overrides startTimerReadStatus() to do nothing, so nothing else touches the instruction list while the IOThread connects. The tests lose only the 5 s status poll; the connect sequence still reads the status once. Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1198 Fix the ShimmerDriverPC connect test hang on Linux CI
API_0000X_ByteCommunicationShimmer3lns0_16_11_w_sd_only, _w_sync_master and _w_sync_slave were excluded from the test task in Feb 2026 as failing on CI. Their CI logs have expired, so the cause cannot be confirmed, but it fits the read-status timer deadlock the previous commit avoids: these classes assert values from the post-connect config read (SD log sync, Bluetooth disabled, sync nodes, trial duration), which a stalled connect never reaches. They now build their device through newShimmerPCWithoutStatusTimer() like the other simulator classes, and run again. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The version no longer lives in the source. The three publishing blocks read -PreleaseVersion (falling back to 'local'), and Release Java API passes it: the version typed in the Run form, or by default the next patch after the last API_v release (0.11.8_beta -> 0.11.9_beta). The API_v tags record what was released, so a release needs no uprev PR and nothing has to be committed to the protected master. Checked locally: with -PreleaseVersion=0.11.9_beta all three POMs and their dependencies on each other are 0.11.9_beta; without it, 'local'. Co-Authored-By: Claude Opus 5.5 <[email protected]>
publish failed on shimmerdriverpc with "multiple artifacts with the
identical extension and classifier ('jar', 'null')": the Shadow plugin
adds ShimmerJavaClass.jar (empty classifier) to components.java, and
Gradle 8.14 (DEV-1013) rejects the clash. The PR dry run skips publish,
so the first real release would have failed there after publishing the
other two. Skip the shadow variant, which publishes the plain jar with
its dependencies in the POM, as 0.11.7_beta was.
Without -PreleaseVersion the version is 'local', and a hand-run
`gradlew publish` would push that to GitHub Packages, where it can
never be overwritten. Publishing to a remote repository now refuses
'local'; publishToMavenLocal is unaffected.
Add job timeouts so a hung Gradle daemon cannot hold a runner for the
default 6 hours.
Checked locally (JDK 11): publishToMavenLocal -PreleaseVersion=0.0.0_test
now succeeds, all three POMs and their cross-dependencies read
0.0.0_test, and shimmerdriverpc is the 140 KB thin jar. Publishing to a
file repository without -PreleaseVersion fails for all three.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1115 Fix shimmerdriverpc publish; refuse a 'local' publish
…Classic, and a release gate Hardware-in-the-loop tests of ShimmerDriver/ShimmerDriverPC on a PC with paired units, through ShimmerPC with its timers on, as Consensys and MATLAB run it. - JavaHilScenariosTest (hilTest task, skipped without -Phil.port, left out of the normal test task): streamRest, midStreamSet, reconnectLoop and inStream. inStream requests each in-stream reply (status, directory, battery) and then overlaps SET_RWC with status and battery requests, the DEV-1262 case. Every API call runs under a watchdog that dumps every thread with its locks. - HilShimmerPC, a test-only subclass, counts in-stream replies by type, decoded statuses and battery readings, and captures every driver log line, without changing what the driver does. The configure step does not write the calibration dump back, so runs leave the units' stored calibration alone. - StreamAnalysis is the Android HIL's class (same rules on both); its tests run in CI. - tools/hil/run-java-hil.ps1 finds each unit's COM port from its MAC, builds once, runs each scenario in its own Gradle run under a time limit, checks every report came back, and with -Report posts the commit status java-hil (full default run, clean pushed commit only). - Release Java API stops unless java-hil passed on the commit, or hil_waiver gives a Jira key; the release notes record which. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ink Windows can't open midStreamSet and inStream now start SD logging as Consensys does (DEV-436): stop streaming, start logging on the quiet link, then stream again. SD logging is still stopped mid-stream. Consensys never writes the RTC mid-stream, so the scenarios no longer do either; that is what caused DEV-1245's out-of-order timestamps on the Shimmer3R. The stream is judged per stretch between the deliberate restarts. Loss within 1 s of an SD logging stop is reported but does not fail, because the unit pauses while it closes its file. Loss near a battery read fails, and loss anywhere else is held to the limit. A connect whose Bluetooth link can't be opened (CONNECTION_FAILED; jssc reports "Port busy") is tried once more, and connectRetries in the report counts those retries. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pattern expected the command as one word, but the driver logs it as "[0x72] GET_STATUS_COMMAND". So every commandTimeouts count was 0, and the "no command timeouts" checks could never fail. Checked against the bench logs so far: it now finds all 14 timeouts in them. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The accel check was skipped when the mean was NaN, so a stream with no calibrated accel passed. A Shimmer3 (3E36, LogAndStream v1.1.7) did exactly that: all 7655 samples NaN after the driver read its calibration dump. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…trying it Every "Port busy" the HIL ever met was another program holding the COM port, never a flaky first connect. Two HILs shared a unit, and that sharing produced the misleading DEV-1276. A retry only lets the run go on while the unit is shared, so its results mean nothing. So a connect is no longer retried. When it fails, the report says why: - the port is open in another program (jssc "Port busy"), which the HIL now sees because HilShimmerPC records the messages the driver prints with consolePrintException; - the link could not be opened for another reason; - the unit did not initialise in time. The README now says that nothing else may use the units during a run, and why a BLE host is the dangerous case. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The connect sequence reads the unit's status part way through. On that reply, ShimmerPC.eventLogAndStreamStatusChanged() moved the state from CONNECTING to CONNECTED, and ShimmerDeviceCallbackAdapter marks every CONNECTED as initialised. So an app waiting for CONNECTED and initialised was told the unit was ready while the battery read, the calibration dump and the inquiry were still queued. Whatever the app did next went out in among that sequence: - streaming started part way through it, and a reply from the connect sequence landed in the stream, which then went silent; - a configure cut the calibration dump short, which left a Shimmer3's calibrated accel NaN in every sample (7 of 9 runs on the HIL). The sequence's own end, inquiryDone() and then isReadyForStreaming(), already moves CONNECTING to CONNECTED and sends NOTIFICATION_SHIMMER_FULLY_INITIALIZED. A status reply now leaves CONNECTING to it. The Shimmer3R simulator answered INQUIRY with the Shimmer3's 8-byte settings, which never parse as a Shimmer3R's 11. Its tests had only ever passed on the early CONNECTED, before the inquiry timed out. It now answers with settings that match its own InfoMem. API_00031 connects to the Shimmer3 simulator, whose sequence reads the status before the inquiry. It checks that CONNECTED comes only after the sequence's end, with nothing but the RTC write still queued. It fails without the fix. Co-Authored-By: Claude Opus 5.5 <[email protected]>
After STOP_LOGGING_ONLY_COMMAND the unit stops sending data packets for about 100 ms while it closes its SD file. A command sent in that pause is answered at once, so its ACK comes straight after the previous ACK's CRC, with no data packet before it. processPacket() only looks for an ACK after a data packet, so it threw the reply away. The command then timed out, and while streaming the timeout clears every queued instruction, so whatever the user asked for next (including Stop Streaming) was silently dropped. The Java HIL found it on a Shimmer3 and a Shimmer3R: up to 4 of 5 status and battery requests sent after a logging stop were lost. processWhileStreaming() now looks for an ACK at the very start of the buffer first. It takes one only while a command waits for its ACK, and only when the bytes after it are what that command expects: - for a GET command, its own in-stream reply (FF 8A, then its response opcode); - for a SET command, a bare ACK whose link CRC checks out. So a 0xFF data byte the parser resyncs onto is not taken for one. It looks at no more than 3 bytes, fewer than processPacket() needs, so nothing else changes. Without a link CRC the case can't arise: the previous ACK isn't processed until the stream resumes, so the next command never goes out during the pause. API_00032 feeds the parser the bytes a logging stop produces, then a status reply, a battery reply, or a SET command's bare ACK in the pause, with 1- and 2-byte CRCs. Without the fix, all six of those cases fail. Two more check that an ACK at the start of the buffer is not taken for a different command. Co-Authored-By: Claude Opus 5.5 <[email protected]>
ShimmerBluetooth guards its instruction list with two locks: the device (its synchronized methods) and the list itself. Every accessor takes them device first, list second. getInstruction(), which the IO thread calls for the next instruction, took the list and then called the synchronized getListofInstructions(), so it took them the other way round. So the IO thread could hold the list while waiting for the device, just as a timer held the device while waiting for the list: - the ACK/response timeout, calling clearAllInstructions() while streaming; - the read-status timer, queuing GET_STATUS_COMMAND. Both then waited for good, and so did every later API call. The Android HIL caught it on a Shimmer3, where stopSDLogging() never returned. It was also the Linux CI hang in DEV-1198, which #325 worked around in the tests only. getInstruction() now reads the head of the list directly, so the IO thread never needs the device while it holds the list. This is Mas's suggestion from her review of #326. API_00033 reproduces the deadlock deterministically. A "timer" thread holds the device the way every synchronized accessor does while the IO thread's real getInstruction() runs; the timer then queues an instruction, which needs the list. Without the fix, the JVM's deadlock detector finds the IO thread blocked in getListofInstructions() and the timer blocked in writeInstruction(), the same stacks as the hardware hang. A second test connects five times with the read-status timer left on, as the product runs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
While a unit streams, four app threads call readBattery() and readStatusLogAndStream() and check the queue length, in tight loops for 60 s, keeping the queue under 8. Each of those calls locks the device and then the instruction list, so one is nearly always under way while the IO thread, holding the list, fetches the next instruction. That is the window DEV-1266's lock inversion needs. No SD logging is involved. The scenario fails if the calls and replies stop for 10 s, or if the JVM finds a deadlock. Either way the events get every thread's stack, and the unit is left alone. On a Shimmer3R (v1.1.21) without the DEV-1266 fix (#334), it deadlocked within 0.3 s in both runs. With the fix, two 60-second runs were clean, with about 670 calls and 0 timeouts each. It runs by default, so the release gate includes it. Until #334 merges, the gate fails on a deadlock. Co-Authored-By: Claude Opus 5.5 <[email protected]>
A Shimmer3 (LogAndStream v1.1.7) sometimes gives up one sample slot to answer GET_VBATT mid-stream: packet n, the battery reply, then packet n+2. That was 4 skips in 33 battery reads. It's the unit, not the host: there were no parser errors, and builds with and without the driver fixes skip at the same rate. Filed for the firmware as DEV-1297. The check charged it as packet loss near a battery read, so midStreamSet failed on the Shimmer3 about one run in three. A gap near a battery read is now excused only when it is exactly one sample and the packet after it arrived within 25 ms of a decoded battery reply. Such a gap is reported, as skippedAtBatteryReplies plus an event naming DEV-1297. Anything else near a battery read still fails. A packet the driver dropped would leave stray bytes, which checkDriverLog() fails on anyway. HilShimmerPC now records when each battery reply is decoded. Checked on 3E36: 4 midStreamSet runs all passed. Two of them reported one skip at a battery reply, and there was no other loss near any read. Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1269: Java HIL v1, the driver on real Shimmer3/3R over Classic, and a release gate
…ter_pause DEV-1291: Take an in-stream ACK that comes with no data packet before it
…nect_sequence DEV-1275: Report a Shimmer ready only once its connect sequence has ended
…t_lock_order DEV-1266: Take the instruction list's two locks in one order
… end DEV-1275 was only caught indirectly: a stream lost in reconnectLoop, or NaN accel when a configure cut the calibration dump short. The HIL now checks it directly. For each connect it notes whether the first CONNECTED state came before NOTIFICATION_SHIMMER_FULLY_INITIALIZED, which the driver sends at the end of its connect sequence. The report counts these as connectedBeforeReady, and any one fails the scenario. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…check DEV-1269: Java HIL checks that every connect reports ready only at the end (DEV-1275)
Three checks the C# and shimmer-logandstream HILs already run on the same bench: - rollover: streams 10 minutes, past the 512 s wrap of the unit's 3-byte timestamp counter, and fails on streamRest's checks or if the raw counter never wrapped. TimestampWraps counts the wraps from the TIMESTAMP channel's raw ticks, which HilSession now records with each sample. - idleStart: leaves the link idle 60 s with the driver's timers on, then fails if the link was lost or a command timed out during the wait, if streaming starts more than 500 ms after startStreaming(), or on the stream checks. - samplingRates: writes 51.2, 256 and 512 Hz in turn the Consensys way, streams 5 s at each and checks the rate, then writes the unit's own rate back. run-java-hil.ps1 runs all eight scenarios by default, so -Report requires the new ones too. TimestampWrapsTest runs in CI. No driver code changes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every reply path retires the command in flight with removeInstruction(0), taking the head of the instruction list to be the command answered. Two things broke that around a stop: - operationPrepare() cleared the list while the STOP was still in flight. The stop handling, which sleeps 400 ms on the first byte after the STOP, then removed the head in the STOP's place: the first SET_INFOMEM of a configuration started straight after stopStreaming(), the chunk that holds the sampling rate. The unit kept its old rate. operationPrepare() now waits, for up to 3 s, until a command already in flight has been answered and retired, then clears the list. mTransactionCompleted is volatile, so the waiting thread sees it. - The stop handling then let the same byte go on to the ACK processing. When that byte was the unit's ACK, it retired the next instruction as well, a command queued after the stop that was never sent. The stop handling now returns once the stop is handled, and logs when it was answered by its ACK. API_00031_ConfigureStraightAfterStopTest runs the IOThread over an in-memory radio. Without the fix the stop-window configuration sends only chunks 0x0080 and 0x0100, and the command queued behind an ACK-first stop is never sent; with it both pass, as does a configuration after the stop has finished. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…res_wrong_instruction DEV-1306: a stop's reply no longer retires the wrong instruction
DEV-1305: Java HIL rollover, idleStart and samplingRates scenarios
ShimmerPC.deepClone() copies a device by serialising it, which takes no lock. On a connected device the timers and the IO thread kept changing the instruction list (and, while streaming, the PC timestamp list and packet queue) during the copy, so deepClone() could throw ConcurrentModificationException and the configure built on it failed. A clone is a copy of the configuration, not of the connection: those three queues are now transient and readObject() gives the copy empty ones. The Java HIL's HilShimmerPC had the same problem with its own log, which the driver's threads append to on every message; its counters are now transient too. Tests: API_00034 clones while a thread queues and retires instructions, and a simulator-connected device while an app reads battery and status; HilShimmerPCTest clones while the driver logs. All three fail on master. Co-Authored-By: Claude Opus 5.5 <[email protected]>
On a Shimmer3 streaming at 256 Hz, about one copy in a hundred threw ConcurrentModificationException in mListofPCTimeStamps, which the IO thread changes for every packet. And the processing thread's packet queue holds packets that can't be serialised, so deepClone() returned null whenever one was waiting. Both tests fail on master. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…lists DEV-1310: deepClone() no longer copies the connection's queues
MAzalya
left a comment
There was a problem hiding this comment.
Reviewed at 03106bf. This looks good to me.
- Lifecycle. The clock starts lazily from the data thread, throttled to one request per 100 ms, and only when it is enabled, the X axis is time and the chart is showing. Each tick stops it on: no sample for 2 s, pause, hidden or iconified, a replaced chart, a non-time X axis, cleared data, or the clock being disabled.
removeAllSignals()(called fromframeClosed()) stops it immediately. A stoppedjavax.swing.Timerleaves the TimerQueue, so there is no leak and no cost when idle. Chart2D's repaint latency is restored on stop. - Locking with #280. The tick takes only the chart monitor.
noteLiveSampleForFrameClockruns undermListofPropertiestoPlotbut only callsinvokeLater, so the list → chart lock order can't be inverted. - Callers that don't enable the clock see no behaviour change.
DEFAULT_LINE_THICKNESSis still 2. setTraceLineThicknessapplies to traces created afterwards, and the javadoc documents that it does not restyle existing traces. That is fine for Advance-API#494, which sets it before any trace exists.
Minor, not blocking:
- Margin vs post-stall lag.
LIVE_VIEWPORT_TRACE_MARGIN_MS(400 ms) is smaller than the lag the clock allows after a data stall. During a stall the window freezes at the newest X. When the buffered data lands below the 1000 ms resync threshold, the window catches up at ≤2× rather than snapping. So a Bluetooth stall longer than 400 ms leaves a blank strip at the left edge, of up to about (stall − 400 ms), for under 0.5 s. A margin of about 1 s would cover it. ThedataStall_test only asserts recovery time. mIsPlotPausedis not volatile but is read by the EDT tick. If pause is set off the EDT, the window can scroll for up to ~200 ms longer before stopping. Benign, butvolatileis cheap.
CI: no checks have run on this PR, because the workflow only triggers for PRs into master. So API_00029_LiveViewportFrameClockTest has not run anywhere yet.
Merge order: once #280 merges, retarget this to master and let CI run before merging. If #280 is squash-merged, rebase this branch first, or #280's original commits will reappear in the diff. Advance-API#494 depends on this PR and can't compile in CI until it is on master.
Also: the hardware result in the description (#280 on its own gave no improvement, and the stutter is frame cadence) means #280's description still says "Fixes the stuttering…". It's worth correcting there so the release notes credit the right change.
…pause flag LIVE_VIEWPORT_TRACE_MARGIN_MS was 2 x VIEWPORT_LATENCY_MS (400 ms), but after a data stall shorter than FRAME_CLOCK_RESYNC_MS the window catches up at <=2x instead of snapping, trailing the newest sample by up to FRAME_CLOCK_RESYNC_MS + VIEWPORT_LATENCY_MS. A Bluetooth stall over ~400 ms therefore left a blank strip at the window's left edge while it caught up. The margin is now FRAME_CLOCK_RESYNC_MS + VIEWPORT_LATENCY_MS (1.2 s). New test dataStall_lagStaysWithinTraceMargin_forStallsUpToResync covers stalls of 300-980 ms and fails with the old margin. mIsPlotPaused is now volatile: the frame clock tick reads it on the EDT and the data thread reads it too, while pause can be set from another thread. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Addressed in 002b674:
All 8 |
|
not really seen improvement, closing this PR |
Summary
Second part of DEV-896. Hardware testing of #280 on a UHD monitor (one GSR trace at 256 Hz) showed no improvement, and a read-only investigation measured why: the per-sample work #280 removes is under 1 ms of CPU per second in that configuration, and the plot's jank comes from frame cadence, not from the data thread. jchart2d repaints from a 50 ms Swing timer (about 20 fps), and Consensys re-anchored the X viewport to the newest sample's timestamp on every sample, so each frame jumped by whatever Bluetooth burst had arrived since the last one. At UHD each jump covers about twice the pixels.
Stacked on
DEV-896(#280) because it edits the same file. Retarget tomasteronce #280 merges, or rebase if #280 is trimmed.Changes (
ShimmerDriverPC,BasicPlotManagerPC)Live X viewport frame clock. One
javax.swing.Timerper plot manager (FRAME_INTERVAL_MS = 16; 33 is the documented fallback) advances the X window by elapsed time on the EDT and slews it toward the newest sample minus aVIEWPORT_LATENCY_MS = 200playout buffer, so bursts no longer appear as jumps. The window is not anchored to wall-clock: the plotted X is device time plus a one-time offset, so it drifts and NTP can step the PC clock; the clock follows the data's own rate instead. Catch-up after a stall is asymmetric (up to 2x real time when behind, 10% when ahead), with a 1 s resync snap. The window never passes the newest sample and never scrolls backwards.removeAllSignals()(the frame-close path). jchart2d's own repaint timer is backed off to 250 ms while the clock runs and restored on stop, so there is exactly one paint per frame and an idle app does not repaint at 60 fps. OneRangePolicyFixedViewportwith two swappedRangeobjects: no allocation per frame.setLiveViewportFrameClockEnabled(true); nothing changes for consumers that do not call it. The per-samplesetXAxisRangeBasedOnXDuration()stays for explicit callers.Per-plot stroke width. New
setTraceLineThickness(float), read bycreateNormalTraceandsetTraceLineStyle.DEFAULT_LINE_THICKNESSstays 2, so reports, NeuroLynQ, event markers and calibration are unchanged. Consensys live plots set 1 px (companion PR): at identity transform a 2560-point UHD paint measured 0.4 ms at 1 px against 3.7 ms at 2 px (drawPolyline); at uiScale 2 the difference is 3.5 vs 4.0 ms.mCurrentXValueis now volatile.Companion
Shimmer-Advance-API
DEV-896_2enables the clock and the 1 px stroke for Consensys live plots and adds a 1.2 s trace margin (LIVE_VIEWPORT_TRACE_MARGIN_MS=FRAME_CLOCK_RESYNC_MS+VIEWPORT_LATENCY_MS, sized for post-stall catch-up) so the trailing window never shows a blank strip. ConsensysApps needs no change. Merge order: this repo first (#280 then this), then the Shimmer-Advance-API PR, since its CI clones this repo's default branch.Testing
API_00029_LiveViewportFrameClockTest(Swing-free, synthetic clock): monotonic window, bounded lag, 600 ms stall recovery within 1 s, 300 ms EDT hiccup, a device 1.5 s behind, a device reset, 4x playback without snaps, 2 Hz data without frozen runs. 7/7 pass.ShimmerDriverPC./gradlew build: 58 + 177 tests, 0 failures.ShimmerDriverAdvancedPC,ConsensysApiandConsensyscompile against this branch.FRAME_INTERVAL_MSto 33 if the EDT saturates; estimates put that at about 4 plots with 8 traces each).🤖 Generated with Claude Code
https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
Generated by Claude Code